(19) 



J 



Europfliscties Patentamt 
European Patent Office 
Office europten des brevets 




(11) 



EP 1 030 267 A1 



(12) 



EUROPEAN PATENT APPLICATION 

published in accordance with Art. 158(3) EPC 



(43) Date of publication: 

23.08.2000 Bulletin 2000/34 

(21) Application number: 98905802.9 

(22) Date of filing: 06.03.1998 



(51) int CI. 7 . G06T 1/00, G06T 1 1/00 

(86) International application number: 
PCT/JP98/00943 

(87) International publication number: 
WO 98/39735 (11.09.1998 Gazette 1998/36) 

SCALAR CORP. 
Tokyo 151-0053 (JP) 

(72) Inventor: UTSUGI, Ryulchl 
Tokyo 108-0072 (JP) 

(74) Representative: 

Manltz, Flnsterwald & Partner 
Postfach22 16 11 
80506 MQnchen (DE) 

METHOD OF CORRECTING FACE IMAGE, MAKEUP SIMULATION METHODl Maktpiid 
METHOD, MAKEUP SUPPORTING DEVIck AND nSSSSS!^ 



(84) Designated Contracting States: 
DEFRGBIT 

(30) Priority: 06.03.1997 JP 5146097 

(71) Applicants: 
• DROC limited 
Tokyo 1084)072 (JP) 



(54) 



(57) New findings such as the highlight theory which 
determine the impression of a face are utilized for the 
technique of modifying a photographed facial image and 
the makeup technique. A method according to the 
present invention includes determining a desirable face 
desired by a person who is subjected to facial modifica- 



tion or makeup, and performing image processing like 
shape-merging on an original facial image or an image 
of a face of the subject person based on the desirable 
face to thereby prepare images of selectable modified 
faces. 
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Description 

TECHNICAL FIELD 

[0001 ] The present invention relates to a technique 
of modifying facial images and a makeup art of putting 
makeup on a face using the modifying technique. 

BACKGROUND OF THE INVENTION 

[0002] The present inventor has been making stud- 
ies, for a long time, on factors which determine the faces 
of persons, particularly facia! impressions, and beauty 
and ugliness, from the viewpoints of plastic surgery and 
cosmetic surgery. The study has provided several new 
findings. One of them is the "highlight theory". Conven- 
tionally, it was a general thought that the components of 
a face, such as eyebrows, eyes, a nose and lips, would 
become significant factors to determine a facial impres- 
sion. The research done by the present inventor how- 
ever shows that the prime factor to determine a facial 
impression is the shading state of a face, i.e., the states 
of a bright area and a dark area, particularly, the state of 
the brightest area or highlighted area. The shading 
state relates to shading on a face which occurs under 
illumination of the natural light and reflects or 
expresses the three-dimensional shape of the face. The 
states of a highlighted area in such shading particularly 
affects a facial impression. 

[0003] The analysis based on this highlight theory 
classifies faces into three general types. As exemplarily 
shown in Figure 5. the shapes of highlighted areas are 
an inverted triangle type [(a) in Figure 5], an oval type 
[(b) in Figure 5) and a clover-leaf type [(c) in Figure 5]. A 
typical beautiful face in each facial type has a good 
shape of the associated highlighted area. With regard to 
an inverted triangle type, for example, no or little distor- 
tion or deformation on its shape is a significant factor in 
giving a good impression on a face. 
[0004] Another finding is a theory on the shapes of 
faces people feel beautiful or preferable. Conventional 
theories on facial beauty include canon theory or a the- 
ory based on golden cut, and the average theory which 
has recently come up. The average theory means a the- 
ory that the face that appears most beautiful among 
those of a certain human race is an average face of the 
individual faces of that race. Faces based on those the- 
ories merely provide general "beautiful faces". In other 
words, those theories cannot give ideal post-makeup 
faces which are makeup aims applicable to makeup 
methods. Unlike those theories, the theory found by the 
present inventor can be put to a practical use in makeup 
methods. 

[0005] According to what has been newly found by 
the present inventor, through the following procedures, 
it is possible in most occasions to find the most beautiful 
or preferable face for each specific person, or the most 
beautiful or preferable face that can be made by 



makeup available to that individual. The fundamental 
theory is such that if faces are produced by combining a 
face of given standards, specifically, a face generally 
considered beautiful or a face preferred by an individual 

5 and the actual face of that particular individual at the 
proper mixing ratios through an image processing tech- 
nique, an Ideal face" which is considered most beautiful 
or preferable by that individual can be found in most 
cases from the foregoing mixed faces. Specifically, one 

10 of model faces, which are. for example, faces of pre- 
ferred talents or actresses or actors, or "beautiful faces" 
originated from the average theory and the like, is 
selected as a desirable face. Based on this desirable 
face, the actual face of the person is subjected to image 

is processing like morphing so that the actual face is 
mixed with the desirable face to approach the latter. 
This process provides a plurality of "modified faces" with 
different mixing ratios or similarity levels with respect to 
the desirable face. In most cases, those "modified 

20 faces" contain an "ideal face" that person considers 
most beautiful or preferable. The data obtained by the 
present inventor shows that persons in 48 cases out of 
50 could find "ideal faces'* they thought most beautiful or 
preferable. In the other two cases, "ideal faces" could be 

25 found by modifying the desirable faces. The mixing ratio 
in every case ranged from 10 to 50%. 
[0006] It is to be noted that one mode! face itself 
should not necessarily be the most preferable face for a 
particular person. That is. the "ideal face" a particular 

30 person can find for herself or himself is a "mocfif ied face" 
which is resulted from the mixing of a model face with 
her or his own face, not the model face itself The actual 
results of the experiments conducted by the present 
inventor show that most subject persons did not prefer 

35 model faces using "beautiful faces" created by the aver- 
age theory or the like. 

[0007] The present invention has been devised to 
effectively utilize the above-described findings and aims 
at using those findings to a technique of modifying, for 
40 example, photographed facial images and to a makeup 
technique. 

DISCLOSURE OF THE INVENTION 

45 [0008] According to the present invention, facial 
images picked up by a still camera, a video camera or 
the like are modified as follows. First, a desirable face is 
determined, and then an original facial image is sub- 
jected to image processing to become closer to this 

so desirable face so that a modified facial image is 
aoquired. 

[0009] Model faces can be used in determining a 
desirable face The faces of preferable talents or 
actresses or actors can be used as model faces. If one 
55 does not have any preferable talent or the like, standard 
good-looking faces acquired from a plurality of beautiful 
faces which have previously been prepared correspond- 
ing to human races or like according to the foregoing 
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average theory may be used. In the foregoing method. It 
may also be arranged that an original facial image is 
subjected to image processing to become doser to a 
desirable face, thereby forming images of a plurality of 
selectable modified faces with different similarity levels 5 
with respect to the desirable face, and then a selection 
is made from those selectable modified faces to acquire 
a modified facial image. This further facilitates acquiring 
a face ideal to the subject person. 
[001 0] It is preferable to use the foregoing highlight 10 
theory in modifying and making an original facial image 
closer to a desirable face. That is, the Key point is to 
make the states of highlighted areas of a facial image to 
be modified closer to those of a desirable face. This can 
provide a more beautiful and preferable face and thus is 
facilitate approaching to a desirable face. 
[0011] This facial image modifying method is an 
application of the foregoing theory on the shapes of 
faces that people think beautiful or preferable. Unlike 
the conventional photograph modification techniques 20 
which mainly depend on a manual work, this method 
can automate the modification work by electronic 
processing. What is more, modification results can give 
greater satisfaction. This method of the present inven- 
tion can be adapted to operational simulation in plastic 2s 
surgery and cosmetic surgery as well as ordinary pho- 
tograph modification. 

[0012] In a makeup simulation method embodying 
the present invention and a makeup method which uses 
this simulation method, a desirable face is determined 30 
first. In the case where a makeup instructor instructs a 
subject person how to put makeup, for example, a desir- 
able face may be determined by asking the subject per- 
son about preferences. Alternatively, model faces may 
be used in determining a desirable face. The faces of ss 
talents, or actresses or actors the subject person like 
can be used as model faces, tf the subject person has 
no preference for any talent or the like, averages of 
good-looking faces which have previously been pre- 
pared corresponding to human races or the like may be 40 
used as model faces. In the former case, a preferred 
desirable face can be selected from model faces using 
the name or the like of the preferable talent or the like as 
an index. In the latter case, a preferred desirable face 
should be selected from model faces of standard beauty 45 
which has previously been input as data. In both cases 
a selection is made while sequentially displaying the 
model faces on a monitor screen or based on the type 
of the face of the subject person. In the latter case, the 
model faces should not necessarily be shown to the so 
subject person. 

[0013] Once the desirable face is decided; then 
images of imagined post-makeup faces based on the 
desirable face or post-makeup faces which are imag- 
ined as preferable are prepared. This involves a process ss 
of mixing the face of the subject person with the desira- 
ble face to make the former face doser to the latter by 
performing image processing like shape-merging on the 



image of the face of the subject person. Then, an ideal 
post-makeup face the subject person desires is deter- 
mined from the imagined post-makeup faces. Specifi- 
cally, a plurality of imagined post-makeup faces with 
different mixing ratios or similarity levels with respect to 
the desirable face are acquired through the foregoing 
image processing, and a preferable selection is made 
from those faces as an ideal post-makeup face within 
the range of the similarity levels which xanie obtained 
by makeup. Thus, the ideal post-makeup face which is 
expected to be finally obtained is given in advance. That 
is, the subject person can know the makeup-finished 
face in a short time. As apparent from the above, the 
significant feature of the makeup method embodying 
the present invention lies in that a final, makeup-finished 
face can be presented in a short time by determining a 
desirable face on which a makeup procedure is based 
and acquiring an ideal post-makeup face with respect to 
this desirable face. In this feature is effectively used the 
above-described theory on the shapes of faces people 
think beautiful or preferable. 

[001 4] When an ideal post-makeup face is decided 
through the foregoing makeup simulation, a makeup 
procedure is derived from this ideal post-makeup face. 
Specifically, a sequence of makeup procedures for 
achieving the ideal post-makeup face, such as selecting 
which portions of eyebrows to be shaved and where to 
draw eyebrows, selecting the lines and ranges for eye 
liners and eye shadows, selecting the colors of eye 
shadows, setting the drawing range -for-a tipstick-and 
separately applying (Afferent foundations, is derived 
from a previously set makeup program. Based on this 
makeup procedure, makeup is applied to the subject 
person. This makes it possible to accurately realize the 
ideal post-makeup face or the makeup state the subject 
person has previously found appropriate, on the face of 
the subject person. That is, makeup which the subject 
person desires can be done freely and in a short time. 
[0015] To realize the foregoing feature of the 
makeup method of the present invention which lies in 
the preparation of an ideal post-makeup face through 
image processing based on a desirable face or a model 
face, it is important to mix the current face of a subject 
person with a model face through image processing to 
get the former face doser to the latter. Tfcat is, it is 
essential to acquire an ideal post-makeup face by mak- 
ing the current face of the subject person closer to the 
reference model face by image processing. The feature 
also requires an effective scheme or standards for mak- 
ing the current face of the subject person closer to the 
ideal post-makeup face by actual makeup. This can be 
accomplished by the foregoing highlight theory. 
[001 6] Based on the highlight theory which teaches 
highlighted areas considerably affect the impression of 
a face and a beautiful and preferable face can be made 
by erasing any unnecessary shade (dark area) to make 
the shapes of the highlighted areas better, it becomes 
relatively easy to get the face of a subject person closer 



3 



5 



EP 1 030 267 A1 



6 



to a model face through image processing and it is also 
easy to make the face of the subject person closer to a 
desirable face by implementing such a makeup as to 
adjust the states of the highlighted areas in association 
with those of the model face (desirable face). In other 5 
words, a more effective scheme of making the face of 
the subject person closer to the reference model face 
can be acquired based on the highlight theory. 
[0017] The image processing for preparing the 
images of imagined post-makeup faces on the basis of 10 
the highlight theory involves a process of making the 
shapes of the eyebrows, the eye lines, the lip line, and 
so forth closer to those of a desirable face within a range 
where modification by makeup is possible, in addition to 
the basic process of getting the states of highlighted is 
areas on a face closer to those of the desirable face 
within a range where modification by makeup is possi- 
ble. A shape-merging scheme or the like can be also 
used in this case. 

[0018] Based on the highlight theory, attention 20 
should be paid to a greatest highlighted area as an 
impressing element of a face in the makeup simulation 
method and makeup method, too. Since makeup about 
highlighted areas is accomplished by separately apply- 
ing different foundations, the types of foundations and 25 
procedures for applying them separately are derived as 
essential factors from an ideal post-makeup face. 
[0019] To accomplish the process, image process- 
ing is performed to divide the ideal post-makeup face 
into a plurality of areas originating from differences 30 
between bright and dark areas like a pattern of contour 
lines. Then, color data is acquired for each segmented 
area. Color data in this case consists of hue and bright- 
ness, and chroma rf necessary, and R, G and B which 
are used in a television system or C. M. Y and K which 
are basic printing colors in a computer system are nor- 
mally used. Based on the color data of the individual 
areas and a previously set foundation program, founda- 
tions to be applied on the individual areas are deter- 
mined. Finally, the thud determined foundations are 
applied on the face of the subject person in accordance 
with the segmented areas. 

[0020] One makeup simulation method according 
to the present invention and a makeup method using 
this simulation method therefore include the following 
steps: (a) determining a desirable face and performing 
image processing on an image of a face of a subject 
person based on the desirable face, thereby forming 
images of imagined post-makeup faces; (b) displaying 
the imagined post-makeup faces, prepared in the step 

(a) , on a monitor screen or the like and selectively deter- 
mining an ideal post-makeup face desired by the sub- 
ject person from the displayed imagined post-makeup 
faces; (c) deriving a makeup procedure from an image 
of the ideal post-makeup face, determined in the step 

(b) , based on a previously s'et makeup program; and (d) 
applying makeup on the face of the subject person in 
accordance with the makeup procedure derived in the 



step (c). 

[0021] Another makeup simulation method embod- 
ying the present invention and another makeup method 
using this simulation method, particularly, a makeup 
simulation method for applying makeup based on dark 
and bright states on a face and a makeup method using 
this simulation method, include the following steps: (e) 
determining a desirable face and performing image 
processing on an image of a face of a subject person 
based on the desirable face, thereby forming images of 
imagined post-makeup faces; (f) displaying the imag- 
ined post-makeup faces, prepared in the step (e). on a 
monitor screen or the like and selectively determining 
an ideal post-makeup face desired by the subject per- 
son from the displayed imagined post-makeup faces; 
(g) performing image processing to segment an image 
of the ideal post-makeup face, determined in the step 
(f), to a plurality of areas in regard to bright areas and 
dark areas; (h) acquiring color data for the segmented 
areas obtained in the step (g); (i) determining founda- 
tions to be applied on individual areas based on the 
color data for those areas, acquired in the step (h), and 
a previously set foundation program; and (j) separately 
applying foundations, determined in the step (i). on the 
face of the subject person in association with the indi- 
vidual areas. 

[0022] The makeup method based on dark and 
bright states may be modified in such a way as to be 
able to print the image of the ideal post-makeup face, 
segmented into a plurality of areas in regard to bright 
areas and dark areas in the step (g), in an actual size of 
the face of the subject person, to cut out individual 
areas from the printed image to prepare a pattern mask 
for foundation, and to execute separate application of 
foundations in the step (k) by using this pattern mask. 
This design can allow different foundations to be sepa- 
rately applied on the associated areas easily and accu- 
rately. That is. separate application of different 
foundations associated with an ideal post-makeup face 
can be carried out easily and accurately. 
[0023] The makeup method which is based on the 
highlight theory may be modified in such a manner as to 
be able to project the image of the ideal post-makeup 
face, segmented into a plurality of areas in regard to 
bright areas and dark areas in the step (g), in an actual 
size on the face of the subject person, and to execute 
separate application of foundations in the step Q) based 
on this projected image. This modification can also per- 
mit different foundations to be separately applied on the 
associated areas easily and accurately. 
[0024] Another makeup method according to the 
present invention which is based on dark and bright 
states includes the following steps: (A) determining a 
desirable face and performing image processing on an 
image of a face of a subject person based on the desir- 
able face, thereby forming images of imagined post- 
makeup faces; (B) displaying the imagined post- 
makeup faces, prepared in the step (A), on a monitor 
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screen or the like and selectively determining from the 
displayed imagined post-makeup faces an ideal post- 
makeup face the subject person desires; (C) performing 
image processing to segment an image of the ideal 
post-makeup face, determined in the step (B), to a plu- s 
rality of areas in regard to bright areas and dark areas; 
(D) acquiring color data for the segmented areas 
obtained in the step (C); (E) determining foundations to 
be applied on individual areas based on the color data 
for those areas, acquired in the step (D), and a previ- 10 
ously set foundation program; (F) printing foundations, 
determined in the step (E) in association with the plural- 
ity of areas segmented in regard to bright areas and 
dark areas in the image processing in the step (C). on a 
thin base material to prepare a transfer film; and (G) js 
transferring foundations, printed on the transfer film pre- 
pared in the step (F). on the face of the subject person 
to thereby apply foundations, associated with the indi- 
vidual areas, on the face of the subject person. 
[0025] A further makeup method according to the 20 
present invention which is based on dark and bright 
states includes the following steps: (H) determining a 
desirable face and performing image processing on an 
image of a face of a subject person based on the desir- 
able face, thereby forming images of imagined post- 25 
makeup faces; (I) displaying the imagined post-makeup 
faces, prepared in the step (H), on a monitor screen or 
the like and selectively determining from the displayed 
imagined post-makeup faces an ideal post-makeup face 
the subject person desires; (K) printing an image of the 30 
ideal post-makeup face in the bright and dark states, 
determined in the step (I), on a thin base material using 
foundations to prepare a foundation transfer film; and 
(L) transferring foundations, printed on the foundation 
transfer film prepared in the step (K). on the face of the ss 
subject person to thereby apply the foundations on the 
face of the subject person. 

[0026] Those methods can repeatedly realize 
makeup for an ideal post-makeup face by carrying out 
simple procedures, such as transferring foundations 40 
from a transfer film accurately corresponding to founda- 
tions for the ideal post-makeup face and properly shad- 
ing off the transferred foundations at the boundaries 
between the individual areas. Thus, desired makeup 
can be carried out more easily and in a shorter time. <5 
[0027] A makeup support apparatus according to 
the present invention for use in the foregoing makeup 
simulation methods and makeup methods comprises 
image storage means for storing images of a plurality of 
model faces; imaging and input means for imaging and so 
inputting a face of a subject person; a monitor for dis- 
playing a necessary image; and processing means 
capable of performing a process of causing an image of 
the face of the subject person to approach a desirable 
face already determined to thereby form images of ss 
imagined post-makeup faces, a process of displaying 
the imagined post-makeup faces on a screen of the 
monitor, and a process of deriving a makeup procedure 



from a determined ideal post-makeup face. 
[0028] It is preferable that the processing means in 
the makeup support apparatus may further perform a 
process of segmenting the determined ideal post- 
makeup face to a plurality of areas in regard to bright 
areas and dark areas, a process of acquiring color data 
for the segmented areas, and a process of determining 
foundations to be applied on individual areas based on 
the acquired color data for those areas. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0029] 

Figure 1 is a diagram for explaining one example of 
image processing in a facial image modifying 
method. 

Figure 2 is a structural diagram of a makeup sup- 
port apparatus according to one carrying-out mode 
of the present invention. 

Figure 3 is an explanatory diagram concerning a 
highlight theory. 

Figure 4 is a diagram for explaining segmentation 
into highlighted areas. 

Figure 5 is an explanatory cfiagram of the types of 
highlighted areas. 

BEST MODE FOR CARRYING OUT THE INVENTION 

[0030] Hereinbelow, carrying-out modes of the 
present invention will described. To begin with, a carry- 
ing-out mode of a facial image modifying method 
according to the present invention will be described. 
Normally, a system which includes a data processing 
apparatus like a personal computer and a monitor con- 
nected to the apparatus is used in implementing the 
facial image modifying method of the present invention. 
An image of a face to be modified or an original facial 
image is put into the data processing apparatus. Then, 
a desirable face is selected from previously prepared 
model faces, e.g., multiple standard faces which have 
been prepared based on the average theory, and the 
original facia) image is subjected to image processing 
based on this desirable face. 
[0031 ] The image processing is executed by a pro- 
gram which employs an image processing scheme tike 
morphing or twining. Specifically, the coordinate values 
of facial constituents, such as eyebrows, eyes, a nose 
and a mouth, of both an original facial image M and a 
desirable face T, and the coordinate values of bright and 
dark patterns of each face are acquired as shown in Fig- 
ure 1, for example, and the original facial image M is 
mixed with and transformed toward the desirable face T 
based on those coordinate values. 
[0032] More specifically, first essential portions, 
such as eyebrows, eyes, a nose, a mouth and the lines 
of highlighted areas, are plotted for each of the original 
facial image M and the desirable face T. For example, a 
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tip portion of the nose is expressed as "0", the eye lines 
as "1-2" and "3-4", and the eyebrows as "5-6" and "7-8". 
Then, those points on the original facial image M are 
shifted toward the corresponding points on the desirable 
face T by a morphing program. The lines of the high- 5 
lighted areas or the like are shifted or their inclinations 
are changed by a twining program. Such image 
processing by shifting points or the like includes a plu- 
rality of phases according to the degree of such shifting. 
That is, selectable modified faces (imagined post- 10 
makeup faces in the case of a makeup method which 
will be described later) corresponding to those phases 
can be obtained. If point shifting is performed every one 
tenth of the distance between corresponding points on 
both facial images, for example, there are ten phases 75 
and there are thus ten imagined post-makeup faces. 
According to the experiments conducted by the present 
inventor, most of the times, subject persons found their 
"ideal faces" (selectable modified faces or imagined 
post-makeup faces) on selectable modified faces at the 20 
second or third phase, i.e., selectable modified faces 
which had approached to their desirable faces by 
degrees of 20 to 30%. That is, while there is a desirable 
face as an image, a face too dose to this desirable face 
comes off the actual "ideal face". 25 
[0033] As the above process forms the images of a 
plurality of selectable modified faces with different simi- 
larity levels with respect to a desirable face, they are dis- 
played on the screen of the monitor to permit the subject 
person to make the best selection, which is treated as a 30 
modified facial image. 

[0034] Now, carrying-out modes of a makeup simu- 
lation method and a makeup method according to the 
present invention will be described. The makeup 
method of the present invention is earned out by using a 3$ 
makeup support apparatus. The makeup support appa- 
ratus of which structure is illustrated in Figure 2 includes 
a data processing apparatus 1 , a storage device 2, a 
monitor 3 and an image sensing unit 4. The data 
processing apparatus 1, which executes various kinds 40 
of data processing required in the makeup method of 
the present invention, is a personal computer or the like. 
The storage device 2 is used to store data of model 
faces which are used in the makeup method of the 
present invention. An internal memory device or an 45 
external memory device is used as the storage device 
2. 

[0035] In an exemplified case where a makeup 
instructor instructs a subject person a makeup proce- 
dure using such a makeup support system, the makeup so 
method proceeds as follows. First, the instructor asks 
the subject person specific questions to determine the 
impression of a face desired by the subject person or a 
desirable face. Model faces are used to make a desira- 
ble face concrete. As model faces are used image data ss 
of faces of popular talents, actresses or actors or the 
like, or standard good-looking faces prepared in associ- 
ation with human races or the like, which are stored in 



the storage device 2. A desirable face is selected from 
the model faces by using an average face for each race, 
each era. each personality, each work or the like or the 
name of a specific talent, or is selected from various 
model faces sequentially displayed on the screen of the 
monitor 3. 

[0036] Once the desirable face is determined, the 
face of the subject person is picked up by the image 
sensing unit 4 and its image is sent into the storage 
device 2. By performing image processing on the image 
of the face of the subject person using the data process- 
ing apparatus 1, the image of the face of the subject 
person and the image of the desirable face are mixed to 
make the former image closer to the latter. In the image 
processing, the original facial image of the face of the 
subject person acquired in the above process is sub- 
jected to the same scheme as employed in the above- 
described modifying method to make the highlighted 
areas, the shapes of the eyebrows, the eye lines, the lip 
line and the like on the face approach those of the desir- 
able face step by step. 

[0037] Figure 3 exemplifies highlighted areas, (a) in 
Figure 3 shows the states of bright and dark areas on 
the original facial image of the face of the subject per- 
son, and (b) in Figure 3 shows the states of bright and 
dark areas made close to those of the desirable face in 
image processing. In the figure, the densely dotted por- 
tions represent relatively dark portions or dark areas 
and a portion surrounded by the dark portions represent 
the brightest area (highlighted area). As apparent from 
this example in which the highlighted area belongs to an 
oval shape, the current face of the subject person is off 
the ideal oval shape and the face of the subject person 
can be made closer to the desirable face which has an 
ideal oval highlighted area by mainly modifying the off 
portions. 

[0038] As this process proceeds, the images of a 
plurality of imagined post-makeup faces with different 
similarity levels with respect to a desirable face are 
formed as described above, and those images are dis- 
played on the screen of the monitor 3 to permit the sub- 
ject person to make a selection. If the subject person 
agrees with any of the displayed images, the selected 
face is set as an ideal post-makeup face. 
[0039] Once the ideal post-makeup face is deter- 
mined, a sequence of makeup procedures about por- 
tions of eyebrows to be shaved and where to draw 
eyebrows, the lines and ranges for eye liners and eye 
shadows, the drawing range for a lipstick and separate 
application of different foundations, is derived from the 
ideal post-makeup face. This sequence is carried out by 
a previously set makeup program. 
[0040] With regard to foundation, in particular, how 
to apply foundations is derived by performing the follow- 
ing processes under the control of a foundation program 
which is included in the makeup program. First, image 
processing is performed to divide the ideal post- 
makeup-face into a plurality of bright and dark areas, 



<EP 1Q302B7A1 I > 



6 



11 



EP 1 030 267 A1 



12 



e.g.. two to four areas This process is exemplified in 
Figure 4. tn the exemplified diagram, the face is seg- 
mented to four areas with different brightnesses (black- 
filled portion, densely-dotted portion, thinly-dotted por- 
tion and white portion). Next, color data tor the seg- 5 
merited areas are acquired. The color data in this case 
are based on R, G and B used in a television system. 
Based on the obtained color data of each area, a foun- 
dation to be applied on that area is determined. 
[0O41] As the makeup procedure needed for mak- w 
ing up the face of the subject person into the ideal post- 
makeup face and cosmetics to be used in the makeup 
procedure are determined in the above manner, actual 
makeup is finally applied to the face of the subject per- 
son based on what has been determined. In utilizing the is 
thus derived makeup procedure and the like in the 
actual makeup, data about the makeup procedure, etc. 
should be displayed on the monitor screen or printed 
out 

[0042] According to a second carrying-out mode of 20 
the present invention, a pattern mask is used to apply 
foundations on the face of a subject person. A pattern 
mask is formed tike a pattern sheet by cutting out indi- 
vidual areas from a printed image of the face, seg- 
mented into a plurality of areas in regard to bright areas 2s 
and dark areas in the above-described image process- 
ing, in the real size of the face of the subject person. 
Then, the pattern mask is placed on the face of the sub- 
ject person and foundations for the individual areas are 
applied over the pattern mask.- : : " : ;:;r^ 

[0043] A third carrying-out mode of the present 
invention employs a projection scheme to apply founda- 
tions on the face of a subject person. Specifically, the 
image of the face, segmented into a plurality of areas in 
regard to bright areas and dark areas in the above- 35 
described image processing, is projected on the face of 
the subject person in real size, and foundations for the 
associated areas are separately applied based on this 
projected image. 

[0044] A fourth carrying-out mode of the present 40 
invention uses a transfer scheme to apply foundations 
on the face of a subject person. Specifically, first, in 
association with a plurality of areas segmented in 
regard to bright areas and dark areas in the above-dis- 
cussed image processing, foundations for the respec- 45 
tive areas are printed in an actual size on a thin base 
material to prepare a foundation transfer film. Alterna- 
tively, images of bright and dark states on the ideal post- 
makeup face determined in the above-described man- 
ner are printed on a thin base material using founda- so 
tions to prepare a foundation transfer film. Then, such a 
foundation transfer film is pressed against the face of 
the subject person to transfer the foundations. Finally, 
the proper shade-off process or the like is performed on 
the transferred foundations at the boundaries between $s 
the individual areas. The foundation transfer film in this 
case does not necessarily correspond to the entire face, 
but a partial transfer f am for the upper eyelids or a por- 



tion around the nose, for example, may be used. 
[0045] According to the fourth carryingout mode of 
the present invention, application of foundations to the 
face of the subject person is accornpfished by directly 
printing the foundations on the face by means of a 
three-dimensional printer or the like. Specifically, an 
image with separately applied foundations, which is 
formed by the above-described image processing, is 
directly spray-printed on the face by a three-dimen- 
sional printer or the like which uses the necessary foun- 
dations as inks. 

Claims 

1. A facial image modifying method comprising the 
steps of determining a desirable face, and causing 
an original facial image to approach an image of 
said desirable face through image processing to 
acquire a modified facial image. 

2. A makeup simulation method comprising the steps 
of: 

(a) determining a desirable face and perform- 
ing image processing on an image of a face of 
a subject person based on said desirable face, 
thereby forming images of imagined post- 
makeup faces; 

(b) displaying said imagined post-makeup 
faces, prepared in said step (a), on a monitor 
screen or the like and selectively determining 
from said displayed imagined post-makeup 
faces an ideal post-makeup face said subject 
person desires; and 

(c) deriving a makeup procedure from an 
image of said ideal post-makeup face, deter- 
mined in said step (b), based on a previously 
set makeup program. 

3. A makeup simulation method for executing makeup 
simulation based on shading states of a face, com- 
prising the steps of: 

(e) determining a desirable face and perform- 
ing image processing on an image of a face of 
a subject person based on said desirable face, 
thereby forming images of imagined post- 
makeup faces; 

(f) displaying said imagined post-makeup 
faces, prepared in said step (e), on a monitor 
screen or the like and selectively determining 
from said displayed imagined post-makeup 
faces an ideal post-makeup face said subject 
person desires; 

(g) performing image processing to segment 
an image of said ideal post-makeup face, 
determined in said step (f), to a plurality of 
areas in regard to bright areas and dark areas; 
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(h) acquiring color data for said segmented 
areas obtained in said step (g); and 

(i) determining foundations to be applied on 
individual areas based on said color data for 7. 
those areas, acquired in said step (h), and a s 
previously set foundation program. 

4. The makeup simulation method according to claim 
2 or 3, further comprising a step of preparing a plu- 
rality of model faces and determining said desirable to 
face based on said model faces. 

5. A makeup method comprising the steps of: 

(a) determining a desirable face and perform- is 
ing image processing on an image of a face of 

a subject person based on said desirable face, 
thereby forming images of imagined post- 
makeup faces; 8. 

(b) displaying said imagined post-makeup 20 
faces, prepared in said step (a), on a monitor 
screen or the like and selectively determining 
from said displayed imagined post-makeup 
faces an ideal post-makeup face said subject 
person desires; 25 

(c) deriving a makeup procedure from an 
image of said ideal post-makeup face, deter- 
mined in said step (b), based on a previously 
set makeup program; and 

(d) applying makeup on said face of said sub- 30 
ject person in accordance with said makeup 
procedure derived in said step (c). 9. 

6. A makeup method for applying makeup based on 
shading states of a face, comprising the steps of: 35 

(e) determining a desirable face and perform- 
ing image processing on an image of a face of 
a subject person based on said desirable face, 
thereby forming images of imagined post- 40 
makeup faces; 

(f) displaying said imagined post-makeup 
faces, prepared in said step (e), on a monitor 
screen or the like and selectively determining 
from said displayed imagined post-makeup 45 
faces an ideal post-makeup face said subject 
person desires; 

(g) performing image processing to segment 
an image of said ideal post-makeup face, 
determined in said step (0. to a plurality of so 
areas in regard to bright areas and dark areas; 

(h) acquiring color data for said segmented 
areas obtained in said step (g); 

(i) determining foundations to be applied on 
individual areas based on said color data for ss 
those areas, acquired in said step (h), and a 
previously set foundation program; and 

(j) separately applying foundations, determined 



in said step (i). on said face of said subject per- 
son in association with said individual areas. 

The makeup method according to claim 6, further 
comprising: 

(k) printing said image of said ideal post- 
makeup face, segmented into a plurality of 
areas in regard to bright areas and dark areas 
in said step (g) in claim 3, in an actual size of 
said face of said subject person; 
(I) cutting individual areas from a printed image 
obtained in said step (k) to prepare a pattern 
mask for foundation; and 
(m) executing separate application of founda- 
tions in said step (j) in claim 6 using said pat- 
tern mask prepared in said step (I). 

The makeup method according to claim 6. further 
comprising: 

(n) projecting said image of said ideal post- 
makeup face, segmented into a plurality of 
areas in regard to bright areas and dark areas 
in the image processing in said step(g) in claim 
3, in an actual size on said face of said subject 
person; and 

(0) executing separate application of founda- 
tions in said step (j) in claim 3 based on said 
image projected in said step (n). 

A makeup, method for applying makeup based on 
bright and dark states of a face, comprising the 
steps of: 

(A) determining a desirable face and perform- 
ing image processing on an image of a face of 
a subject person based on said desirable face, 
thereby forming images of imagined post- 
makeup faces; 

(B) displaying said imagined post-makeup 
faces, prepared in said step (A), on a monitor 
screen or the like and selectively determining 
from said displayed imagined post-makeup 
faces an ideal post-makeup face said subject 
person desires; 

(C) performing image processing to segment 
an image of said ideal post-makeup face, 
determined In said step (B), to a plurality of 
areas in regard to bright areas and dark areas; 

(D) acquiring color data for said segmented 
areas obtained in said step (C); 

(E) determining foundations to be applied on 
individual areas based on said color data for 
those areas, acquired in said step (D), and a 
previously set foundation program; 

(F) printing foundations, determined in said 
step (E) in association with said plurality of 
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areas segmented in regard to bright areas and 
dark areas in said image processing in said 
step (C). on a thin base material to prepare a 
foundation transfer film; and 

(G) transferring foundations, printed on said s 
foundation transfer film prepared in said step 
(F), on said face of said subject person to 
thereby apply foundations, associated with said 
individual areas, on said face of said subject 
person. 10 

1 0. A makeup method comprising the steps of: 

(H) determining a desirable face and perform- 
ing image processing on an image of a face of is 
a subject person based on said desirable face, 
thereby forming images of imagined post- 
makeup faces; 

(I) displaying said imagined post-makeup 
faces, prepared in said step (H), on a monitor 20 
screen or the like and selectively determining 

an ideal post-makeup face desired by said sub- 
ject person from said displayed imagined post- 
makeup faces; 

(K) printing an image of said ideal post-makeup 25 
face, determined in said step (0. on a thin base 
material using foundations to prepare a foun- 
dation transfer f3m; and 
(L) transfening foundations, printed on said 
foundation transfer film prepared in said step 30 
(K). on said face of said subject person to 
thereby apply said foundations on said face of 
said subject person. 



a determined ideal post-makeup face. 

The makeup support apparatus according to daim 
12, wherein said processing means further per- 
forms a process of segmenting said determined 
ideal post-makeup face to a plurality of areas in 
regard to bright areas and dark areas, a process of 
acquiring color data for said segmented areas, and 
a process of determining foundations to be applied 
on individual areas based on said acquired color 
data for those areas. 

A foundation transfer film for use in separate appli- 
cation of foundations, said foundation transfer film 
having foundations printed on a thin base material 
in regard to bright areas and dark areas on a face. 



11. The makeup method according to any one of claims 3$ 
5 to 10, further comprising a step of preparing a 
plurality of model faces and determining said desir- 
able face based on said model faces. 

12. A makeup support apparatus for use in the makeup *o 
simulation method as recited in any one of claims 2 

to 4 or in the makeup method as recited in any one 
of claims 5 to 1 1 . comprising: 

image storage means for storing images of a 45 
plurality of model faces; 
image pick-up and input means for imaging and 
inputting a face of a subject person; 
a monitor for displaying a necessary image; 
and & 
processing means capable of performing a 
process of causing an image of said face of 
said subject person to approach a desirable 
face already determined to thereby form 
images of imagined post-makeup faces, a ss 
process of displaying said imagined post- 
makeup faces on a screen of said monitor, and 
a process of deriving a makeup procedure from 
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